The present invention relates to a method of multiplexing data for multiplexing and transmitting data such as digital pictures, audio, digital data and the like, a method of decoding for reproducing the multiplexed data, and a device for practicing these methods. More particularly, the present invention relates to a method of multiplexing data, which processes picture data by each object, i.e., a picture is an aggregate of plurality of objects, a method of reproducing the multiplexed data, and a device for practicing these methods. These methods and the device contributes to reduce data quantity and increase an efficiency of data processing.
Recently, digital pictures and audio service are commercially used in broadcasts, CATVs and the like thanks to advancement of compression technique. Regarding the transmission of digital pictures, audio and the like, not only the compression technique but also a method of multiplexing data must be seriously discussed.
The international standard method of multiplexing data, i.e., MPEG 2 (ISO/IEC JTC1/SC29 N801, ISO/IEC CD 13818-1: Information technology-Generic coding of moving pictures and associated audio: Systems, 1994.11) is described hereinafter highlighting the multiplexing data method, and the multiplexed data reproducing apparatus with reference to the attached Figures.
FIGS. 20 and 21 depict a conventional method of multiplexing data of MPEG 2 standard. FIG. 22 is a block diagram depicting a multiplexed-data reproducing apparatus that decodes the multiplexed-data through the multiplexing method of MPEG 2 standard. In FIG. 22, a separator 161 comprises a buffer 1611 and a CPU 1612. A synchronizing controller 162, a picture decoder 163 and an audio decoder 164 are the elements of the reproducing apparatus. FIG. 23 is a flowchart depicting an operation of the CPU 1612 of the separator 161.
An operation of the conventional method of multiplexing data and the multiplexed-data reproducing apparatus according to MPEG 2 standard is hereinafter described.
Picture data and audio data are compressed into PES packet, namely, in the case of picture data, each xe2x80x9cframexe2x80x9d that is a certain number of samples is compressed, then one frame or a plural frames bundled forms a packet called as xe2x80x9cPES packet.xe2x80x9d In the case of audio data, a certain number of samples (in MPEG standard, the sample of audio data is also called xe2x80x9cframexe2x80x9d), for instance, xe2x80x9c1024xe2x80x9d is compressed and then forms a PES packet same as the case of picture data.
FIG. 21 outlines a format of the PES packet. The PES packet is headed by a header, which includes the following information:
Stream ID: identify whether the data area following the header contains picture data, audio data or other data;
Time information for synchronized reproduction pictures with audio data and the resultant data;
DTS (decoding time stamp, decode time information);
PTS (presentation time stamp, presentation time information). The PES packet is divided into a plural TS packets (each TS packet comprises 188 bytes) before being transmitted.
FIG. 20 outlines a format of the TS packet. The TS packet is headed by PID (packet identification) that is a proper number (packet No.) of each PES packet. The same PES packet has the same PID. The TS packet sends a header followed by data as shown in FIG. 20(b). The PID of TS packet is sent as a part of the header as shown in FIG. 20(c).
In the data area of TS packet, PSI (program specific information) can be sent in addition to PES packet as shown in FIG. 20. PSI includes information about selecting a program. FIG. 20(d) depicts a structure of program association table (PAT), which is one of PSI, and FIG. 20(e) depicts a structure of program map table (PMT).
The PAT is assigned PID=0, where a PID of PMT that contains program No. and its content is described. Respective PIDs of a program No. indicating a program No. to be transmitted, picture data PES packet, audio data PES packet and data PES packet are included in PMT. This PMT is multiplexed and transmitted.
FIG. 22 depicts an example of a conventional multiplexed-data reproducing apparatus which receives and reproduces the data multiplexed through the above method.
The multiplexed data received by the conventional apparatus is firstly stored in the buffer 1611. The CPU 1612 operates following the flowchart shown in FIG. 23. First, search a PAT of PID=0 in the transmitted data (step 171.) Then, out of the PAT, search a PID of PMT of a desired program No. set from outside (step 172.) Next, pick out a packet having the PMT""s PID obtained from the PAT out of the multiplexed data (step 173.)
After this pick-out, PIDs of picture data PES, audio data PES and data PES packet are stored (step 174.) Finally, pick out a necessary TS packet for reproducing the program by identifying a TS packet""s PID out of the received data (steps 175 and 176).
PID is not described in PAT, and the table of PAT does not contain both of a program No. and PID of PES, but the tables of PAT and PMT are used because it is necessary to avoid a long size table when a multiple programs are multiplexed. This structure can deal with a partial change of a program with ease and flexible by altering a content of PMT.
The picture packet and audio packet out of TS packets are sent respectively to the decoders 163 and 164. The synchronizing controller 162 picks out time information such as multiplexed PTS, DTS and the like from PES packet, and controls a timing of synchronizing reproduction by the decoder. The decoders 163 and 164 reproduce pictures in step with synchronization with audio data, as well as display the resulted pictures.
The above structure, however, produces the following various problems. In picture encoding, independent object is coded, i.e., each component that constitutes a picture, e.g., a background, a person, and a moving body, is individually coded. This object coding method now draws attention. Since the object coding encodes individual body, an editing such as replacing or removing a specific body can be done with ease, also a specific item can be searched easily.
The conventional multiplexing method, however, handles only a concept of xe2x80x9cframexe2x80x9d, and thus it cannot handle a picture with a concept of xe2x80x9cobjectxe2x80x9d. Another method to overcome this disadvantage can be discussed. That is to say, the prior art is extended, and a picture is taken to be a group of objects instead of a frame, and a PES packet is formed to an individual object. In other words, a method that assigns other PIDs to each object can be studied. However, when a large number of objects constituting a picture are required, this method must employ a complicated multiplexing apparatus that can deal with increased kinds of packets. Further, when an individual object is encoded, objects must be sometimes overlapped and displayed; however, the conventional method only has a synchronizing function using a time base, but cannot assign a display location in a picture to a receiver side.
The present invention addresses the above problems and aims to provide a method of multiplexing data and a multiplexed data reproducing apparatus that overcome the above problems.
A method of multiplexing data processes digital data on an individual object base, and has a technique to set a hierarchical structure in the object. More specifically, the method multiplexes digital data xe2x80x9cp1xe2x80x9d and xe2x80x9cd1xe2x80x9d into a first object data xe2x80x9co1xe2x80x9d. Further, make a digital data xe2x80x9cp2xe2x80x9d as a second object data xe2x80x9co2xe2x80x9d. The first and second object data xe2x80x9co1xe2x80x9d and xe2x80x9co2xe2x80x9d are treated as an individual object, and these two data are further multiplexed, whereby an upper class third object data xe2x80x9co3xe2x80x9d is obtained. The data can be processed using either this third object data in the upper class or the first and second object data in the lower class.
Another structure is available, i.e., multiplex digital data xe2x80x9cp1xe2x80x9d and xe2x80x9cd1xe2x80x9d to produce the first object data xe2x80x9co1xe2x80x9d. This first object data xe2x80x9co1xe2x80x9d is packetized into a first packet data xe2x80x9cop1xe2x80x9d. The second digital data xe2x80x9cd2xe2x80x9d is made into the second object data xe2x80x9co2xe2x80x9d, which is packetized into a second packet data xe2x80x9cop2xe2x80x9d. Then these packet data are further multiplexed to produce a third packet data xe2x80x9cop3xe2x80x9d. A packet No. that identifies each packet is given to the packet data xe2x80x9cop1xe2x80x9d and xe2x80x9cop2xe2x80x9d respectively, whereby an individual object is assigned a packet No. Data can be thus processed with ease on individual object base. Further, the data can be processed as an upper class data that integrates the first and second objects into the third packet data xe2x80x9cop3xe2x80x9d.
The present invention treats a picture as an aggregate of plural objects, and processes as well as multiplexes picture data on each object base into multiplexed data, where a method proposed by the present invention comprises the steps of:
(a) dividing a picture into plural master objects and identifying themselves as master objects;
(b) dividing the identified master object into further small slave objects and identifying themselves as slave objects, thereby dividing the master object into plural hierarchical classes;
(c) packetizing the identified master object and/or the slave object into a packet, and assigning a packet No. to each packet; and
(d) adding the data of hierarchical classes as hierarchical information to the multiplexed data.
The present invention also provides a method of reproducing the data multiplexed through the above method, and a multiplexed data reproducing apparatus using the method thereof
Since a packet number is assigned to respective objects, data can be processed on an individual object base with ease. Therefore, a target object can be easily picked out, reproduced, or edited. At the same time, data quantity can be reduced thanks to hierarchical division. A master object in an upper class and a slave object in a lower class can be separately processed upon request.
Further, time data of starting a reproduction and location data where the object is reproduced, are multiplexed in the data, whereby the object can be reproduced based on the time data and location data. As a result, processing efficiency can be improved.